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Abstract Body 
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Background / Context: 

Description of prior research and its intellectual context. 


As a result of the 1998 reauthorization of Head Start, the Department of Health and Human 
Services conducted a national evaluation of the Head Start program. The goal of Head Start is to 
improve the school readiness skills of low-income children in the United States. There is a 
substantial body of experimental and correlational research that has found associations between 
the quality of children’s early childhood classrooms and their subsequent academic success 
(Pianta, Barnett, Burchinal, & Thornburg, 2009; Raver et ah, 2008; 2009; 2011; Zaslow et ah, 
2010). A particularly important part of evaluating the effectiveness of Head Start is 
understanding whether the program improves the quality of participants’ early learning 
environments. Results from the original analysis of the classroom quality data from the Head 
Start Impact Study (U.S. Department of Health and Human Services, 2010) show that 
randomization to Head Start lead to improvement in the quality of care children received. 
However, these results are obfuscated by the way in which missing data were handled. The 
apparent improvements in classroom quality may be due to greater access to formal care settings, 
greater access to higher quality formal care settings, or more likely to a combination of the two. 

Purpose / Objective / Research Question / Focus of Study: 

Description of the focus of the research. 


To address this problem, the primary objectives of this research are to address the challenges 
faced when estimating impacts of treatment random assignment on classroom quality within the 
Head Start Impact Study, to explore how different solutions influence the impact estimates, and 
to offer methodological solutions. That is, the goal of this research is to be able to accurately 
answer the following questions: What are the impacts of random assignment to Head Start on 
measures of classroom quality? Do children who are randomly assigned to Head Start receive 
higher/better quality care than children who are randomly assigned to the control group? 

However, there are several challenges to answering these questions accurately. The largest 
threat to estimating unbiased impacts is missing data on the dependent variable, classroom 
quality. We expect that missing data on classroom quality are endogenous to treatment random 
assignment. Previous analyses indicate that random assignment to the invitation to participate in 
Head Start increased children’s enrollment in formal care arrangements. Since a classroom 
quality rating can only be collected when a child is in formal care, “missingness” due to being in 
parental care is expected to be associated with treatment random assignment. Quality data are 
also missing if the child was in a formal care setting that should have been evaluated but was not 
observed because the observer was not granted access. As Head Start centers had knowledge of 
the study, it is expected the data collectors were more likely to gain access to Head Start centers. 
Data are also missing for children missing information on their type of child care arrangement. 
Data are less likely to be missing for this reason among children randomized to Head Start 
because they were more likely to be in Head Start and have their care setting known. If 
“missingness” is indeed correlated with treatment random assignment, we cannot estimate 
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unbiased impacts of treatment random assignment on quality without first accounting for this 
association. Yet determining the best way of handling missing data is a challenge. 


Setting: 

Description of the research location. 

The Head Start Impact Study sample was designed to be nationally representative of 3- and 4- 
year-olds attending Head Start programs in the United States and included children in 22 states. 
Observations of classroom quality occurred in the child’s primary care setting, including Head 
Start centers, other public and private center-based care facilities, and family child care homes. 

Population / Participants / Subjects: 

Description of the participants in the study: who, how many, key features, or characteristics. 


This research uses data from the Head Start Impact Study and includes 4,440 3- and 4-year-old 
children who were randomly assigned off a waitlist to either receive an invitation to participate 
in Head Start services or to the control group. The sample included 2,447 three-year-olds and 
1,993 four- year-olds. Children were enrolled in 930 different centers and 1,632 classrooms. They 
were from 81 Head Start grantees and 351 Head Start programs. Children were randomized to 
treatment or control based on 202 center groups. A total of 2,644 children were randomized to 
receive Head Start services and 1,796 were randomized to the control group. Children in the 
control group could enroll in other center-based programs or family child care, or could stay at 
home with a parent, relative, or other caregiver (to be referred to as parental care). However, 
there was crossover of treatment assignment as 195 children randomly assigned to the control 
group were in Head Start and not all children assigned to Head Start took up the treatment. 

Missing data on the classroom quality outcome measure is a serious threat to our ability to 
accurately estimate intent to treat impacts of randomization to Head Start on classroom quality. 
Classroom quality data is missing for 814 children who were in parental care, 601 children who 
were in formal care but whose classroom was not observed, and 601 children who were missing 
data on their type of child care setting and an observation. 

Intervention / Program / Practice: 

Description of the intervention, program, or practice, including details of administration and duration. 

Children were randomly assigned to receive Head Start services or to a control group. The 
control group could enroll in other early childhood programs but did not have access to Head 
Start. As Head Start is based on a “whole child” model, children randomly assigned to the Head 
Start group had access to a set of comprehensive services including preschool education, 
medical, dental, and mental health care, nutrition services, and parental involvement activities. 

Research Design: 

Description of the research design. 


Random assignment occurred prior to the beginning of the 2002-03 school year. Children were 
randomly assigned to Head Start within center groups rather than centers because of the small 
size of many centers. Thus small centers were combined with nearby centers into center groups. 
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The Head Start Impact Study was designed to examine impacts of randomization to Head Start 
separately for 3- and 4-year-olds in order to determine if program impacts vary by age. Thus the 
analyses for this poster will be done separately for 3- and 4-year-olds. Data collection began 
during the fall of 2002, after random assignment. Classroom quality was measured during the 
spring of 2003. 

Data Collection and Analysis: 

Description of the methods for collecting and analyzing data. 


The current research utilizes data collected during the first year of the longitudinal Head Start 
Impact Study. The outcome data of interest, classroom quality, were collected in the spring of 
2003. The Early Childhood Environment Rating Scale (ECERS-R; Harms, Clifford, & Cryer, 
1998) is an observation tool used to measure classroom quality in center-based early childhood 
programs. Items and subscales assess the quality of space, materials, and experiences including 
language interactions between teachers and children. In the current research, only the total mean 
ECERS-R score is used. The Eamily Day Care Rating Scale (EDCRS; Harms & Clifford, 1989) 
is an analogous observation tool to the ECERS-R that is used in family child care settings. The 
mean total EDCRS score was used instead of an ECERS-R score for children in family child 
care. The ECERS-R/EDCRS was completed by independent researchers who observed the 
children’s care settings and is scored on a scale of 1 to 7. 

The original analyses of the Head Start Impact Study created classroom quality threshold 
variables on which to estimate impacts of random assignment. One threshold variable was 
“ECERS-R/FDCRS score of 5 or higher”. Children whose care arrangement received a 5 or 
higher on the ECERS-R/EDRCS received a one on this threshold variable, and all other children 
received a zero, including those with missing data. Thus one problem with this method is that it 
does not differentiate low quality and missing data. Another problem is that it does not use the 
full variance of the continuous quality data. 

In our analyses, we first estimate the impact of treatment random assignment on each of the three 
types of missing data using fixed effects for center groups. We estimate these impacts separately 
for 3- and 4-year-olds and for the full sample in order to demonstrate that “missingness” is 
endogenous to random assignment status and is a problem for estimating unbiased intent to treat 
impacts on quality. 

Y missing = Bo + fi;Treatment -i- B 2 ^CQntQV groups + e 

Next, we estimate the impacts of treatment random assignment on classroom quality using a 
multinomial logit model. We created nine mutually exclusive groups of children based on 
quality score and type of “missingness” and estimated the joint impact of randomization to Head 
Start on movement into formal care, observable care, and higher quality care. 

Pr(Y quality) = Bq + fi/Treatmeut -i- fi 2 Zcenter groups + e 

Our second analytic approached relied on the use of Tobit models. Tobit models are typically 
used with censored or truncated data but have also been used in cases in which the censoring is 
due to missing data due to issues of selection. We use Tobit models to parse the estimated 
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impact of treatment random assignment into the dichotomous impact on use of formal care and 
the continuous impact on quality of care. 

Findings / Results: 

Description of the main findings with specific details. 


Results show that treatment random assignment significantly predicts all three types of missing 
data. Randomization to Head Start significantly predicts lower rates of missing data on 
classroom quality due to (1) being in parental care, (2) being in a formal care setting that was not 
observed, and (3) missing data on the focal care arrangement. (Please insert table 1 here.) 

Results also of the multinomial logit model indicates that random assignment to Head Start 
compared to the control group is associated with increases in access to formal care and increases 
in quality of formal care (Please insert table 2 here). For example, the odds for children 
randomized to Head Start to be in formal care with a score on the ECERS between 5 and 6 
instead of parent care is 28 times as high as the same odds for children randomized to the control 
group. Moreover, 70% of Head Start compared to 24% of control group is predicted to be in 
formal care, a difference of 46 percentage points. And 48% of Head Start compare to 6% of 
control group are predicted to be in high quality care (defined as a 5 or higher on the 
ECERS/EDCRS), a difference of 42 percentage points. 

Results of the Tobit analyses indicate that random assignment to Head Start increased the 
probability of having an observed quality score by 48 percentage points. Additionally, 
conditional on having an observed quality score, random assignment to Head Start increases the 
quality of care by 2 points (Please insert table 3 here). 

Conclusions: 

Description of conclusions, recommendations, and limitations based on findings. 


These results confirm that the three types of missing data described above are indeed endogenous 
to treatment random assignment. Thus, methodological approaches must be used to handle 
missing data in order to obtain unbiased estimates of impacts of random assignment to Head 
Start on classroom quality. The results of the multinomial logit and Tobit analyses are similar 
and suggest that a portion of the impact of randomization to Head Start on classroom quality in 
the initial analyses (U.S. Department of Health and Human Services, 2010) is due to being in 
formal care and a portion is due to being in higher quality care. 

Next steps for this work will focus on using multiple imputation and non-response probability 
weighting to estimate the impacts of random assignment to Head Start on quality. We will 
compare those sets of results with those reported here. Our goal is to find the most appropriate 
method to estimate unbiased impacts on quality given the high amount of missing data on the 
measure of quality while utilizing the continuous quality data which will allow for greater 
precision of final impacts when examining treatment impact variation by classroom quality on 
children’s cognitive, social-emotional, behavioral, and health outcomes. Determining the most 
appropriate methods for handling missing data is important to both understand the true impacts 
of treatment random assignment on quality and to be able to correctly model how variation in 
quality influences impacts of treatment random assignment on child school readiness skills. 
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Appendix B. Tables and Figures 

Not included in page count. 


Table 1. Impacts of random assignment on the three types of missing data 



b 

S.E. 

Effect Size 

3- Year-Olds 




In Parental Care 

-0 29*** 

0.02 

-0.62 

Missing ECERS-R/FDCRS but in formal care 

-0.10*** 

0.01 

-0.26 

Missing Type of Care and Observation 

-0 11*** 

0.01 

-0.28 

4- Year-Olds 




In Parental Care 

-0.25*** 

0.02 

-0.52 

Missing ECERS-R/FDCRS but in formal care 

-0.09*** 

0.01 

-0.24 

Missing Type of Care and Observation 

-0.12*** 

0.02 

-0.29 

3- and 4- Year-Olds 




In Parental Care 

-0 27*** 

0.01 

-0.58 

Missing ECERS-R/FDCRS but in formal care 

-0.09*** 

0.01 

-0.25 

Missing Type of Care and Observation 

-0 11*** 

0.01 

-0.28 


*** p <.001 


Table 2. Impacts of random assignment on quality: Multinomial logit results 



Estimate 

S.E. Odds Ratio 

Formal Care, ECERS/FDCRS <3 

P44*** 

0.23 

4.21 

Formal Care, ECERS/FDCRS 3 to 4 


0.17 

6.71 

Formal Care, ECERS/FDCRS 4 to 5 

2.61*** 

0.14 

13.55 

Formal Care, ECERS/FDCRS 5 to 6 

3.33*** 

0.13 

27.86 

Formal Care, ECERS/FDCRS 6 to 7 

3.64*** 

0.21 

37.99 

Formal Care, Missing Observation 

1.08*** 

0.13 

2.94 

Missing Type of Care & Observation 

Q g4**=H 

0.13 

2.31 

Wald (Chi-Square) 

937.40*** 



Note: Parent Care is the reference group 




*** p<.001 




Table 3. Impacts of random assignment on quality: Tobit results 



Treatment 

S.E. 

X2 


Coefficient 


Main Effect 

4.691*** 

0.134 

1796.17*** 

Marginal Effects 




Probability of observed quality score 

0.484*** 

0.013 


Quality score given observed quality 

1.970*** 

0.054 



*** p <.001 
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